Skip to content

fix: from_dict must not deserialize into the caller's dictionary - #13616

Closed
duoduohappy wants to merge 1 commit into
deepset-ai:mainfrom
duoduohappy:fix/from-dict-copy
Closed

duoduohappy wants to merge 1 commit into
deepset-ai:mainfrom
duoduohappy:fix/from-dict-copy

Conversation

@duoduohappy

@duoduohappy duoduohappy commented Nov 7, 2026 •

Copy link
Copy Markdown

Related Issues

Proposed Changes:

default_from_dict copies init_parameters before replacing serialized sub-objects with their instances. Its own comment says why:

Copy so that replacing serialized sub-objects (Secret/ComponentDevice/nested components) with their deserialized instances below does not mutate the caller's data dict in place. Without this, a second deserialization of the same dict would receive already-parsed objects instead of their serialized form.

Sixteen components do their own deserialization into data["init_parameters"] before calling it, so that copy protects nothing. They all follow the same shape:

init_params = data.get("init_parameters", {})     # reference, not a copy
init_params["splitting_function"] = deserialize_callable(...)   # writes into the caller's dict
return default_from_dict(cls, data)               # the copy inside is now pointless

Grouped by what ends up in the caller's mapping:

Kind of handle Components
callable DocumentSplitter, MockTextEmbedder, MockDocumentEmbedder, OutputAdapter, FunctionHook, LLM, Agent (streaming_callback), ChatPromptBuilder, FallbackChatGenerator
enum DocumentWriter (DuplicatePolicy), InMemoryBM25Retriever and InMemoryEmbeddingRetriever (FilterPolicy)
component / dataclass Agent (chat_generator, tools), LLM, FallbackChatGenerator, ToolResultOffloadHook (store), ConfirmationHook (confirmation_strategies), ComponentTool, PipelineTool

Three consequences, all covered by the new test:

  1. the mapping handed in comes back modified — it holds live objects where the serialized handles were,
  2. json.dumps on it now raises TypeError, so the payload can no longer be persisted,
  3. deserializing the same payload a second time raises (KeyError / TypeError / AttributeError).

Fix: deserialize into a copy, and hand default_from_dict a fresh envelope that carries it.

For the two tool classes the change is smaller, because they build the instance directly from the mapping (cls(**inner_data)) rather than through default_from_dict, so copying the mapping is enough.

Scope of the sweep. I scanned every from_dict implementation in the repository (AST, tracking one level of aliasing through data.get(...) / data[...]) for writes into the caller's mapping before any rebinding to a fresh dict. It reported 17 sites; the one left out is Tool, which #13607 handles. The same defect has already been fixed one component at a time — MetadataRouter (#13421), MockChatGenerator (#13471), ListJoiner (#13563), all merged — so this closes the remaining instances rather than adding a fourth one-off. Tool is the last one left and is already claimed by #13607.

How did you test it?

New file test/core/test_from_dict_does_not_mutate.py parameterises four components over the three invariants above (mapping unchanged, still JSON-serializable, second from_dict succeeds). Four cases cover the enum and nested-dataclass shapes: DocumentWriter, InMemoryBM25Retriever, InMemoryEmbeddingRetriever, ChatPromptBuilder.

I checked the test actually fails without the fix — reverting only haystack/ and re-running the equivalent probe gives:

DocumentWriter: mutated=True, json=TypeError, second=KeyError
ChatPromptBuilder: mutated=True, json=TypeError, second=TypeError

and after the fix:

DocumentWriter: mutated=False
ChatPromptBuilder: mutated=False

Existing suites for the touched components: 297 passed, 36 failed, and the same 36 fail on the unmodified tree (git stash baseline, identical set) — they need optional test dependencies (transformers, sentence-transformers) that are not installed here. No regressions.

ruff check and ruff format --check pass on the changed files.

Notes for the reviewer

  • Each site gets the same four-line comment; it can be trimmed to one line if you prefer, but the reason is not obvious from dict(...) alone.
  • ChatPromptBuilder.from_dict reads data["init_parameters"] rather than .get(...), so a payload without that key still raises KeyError as before — behaviour unchanged.
  • There is no shared helper for this yet. If you expect more components to appear, a default_from_dict-adjacent helper that takes the mapping and returns the envelope would remove the footgun; I did not add one because it would touch every component in this PR and make the diff harder to review.

…s serialized

`default_from_dict` copies `init_parameters` before replacing serialized
sub-objects with their instances, so that a second deserialization does not
receive already-parsed objects. Several components did their own
deserialization into `data["init_parameters"]` before calling it, which defeats
that copy: the caller's mapping ended up holding live callables, enums or
components, so it could no longer be serialized with `json.dumps` and a second
`from_dict` raised.

Deserialize into a copy in those components, and hand `default_from_dict` a
fresh envelope that carries it.

The same defect was previously fixed one component at a time (MetadataRouter
deepset-ai#13421, MockChatGenerator deepset-ai#13471, ListJoiner deepset-ai#13563, Tool deepset-ai#13606). This covers
the remaining instances found by scanning every `from_dict` implementation in
the repository for writes through `data.get("init_parameters", {})` and
`data["data"]`.
@duoduohappy
duoduohappy requested a review from a team as a code owner November 7, 2026 15:43
@duoduohappy
duoduohappy requested review from davidsbatista and removed request for a team November 7, 2026 15:43
@vercel

vercel Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

@duoduohappy is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@CLAassistant

CLAassistant commented Nov 7, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@HaystackBot

Copy link
Copy Markdown
Contributor

Hi @duoduohappy, thanks a lot for your contribution! 🙏

We noticed that the Contributor License Agreement (CLA) check (license/cla) hasn't passed yet, so we've temporarily moved this PR to draft and paused the review assignment.

To get your PR reviewed, please sign the CLA via the link in the license/cla check below (or in the CLA bot comment). As soon as the check turns green, this PR will automatically be marked ready for review again and a reviewer will be re-assigned.

@HaystackBot
HaystackBot removed the request for review from davidsbatista November 7, 2026 16:50
@HaystackBot HaystackBot added the cla-pending PR is in draft until the contributor signs the CLA label Nov 7, 2026
@HaystackBot
HaystackBot marked this pull request as draft November 7, 2026 16:51
@sjrl

sjrl commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Thanks for the PR! Our recommendation is to copy the dict yourself (e.g. copy.deepcopy(data)) before passing it to from_dict, so we do not consider this something we need to fix per component. We have merged a few of these in the past, but we are going to stop accepting them going forward, so I will close this one.

@sjrl sjrl closed this Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-pending PR is in draft until the contributor signs the CLA topic:tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants